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ported by modern versions of the GNU 
debugger (gdb) on FreeBSD. Some are only 
supported in the latest version of gdb (8.3 
at the time of writing), while others are 
available in older versions. By John Baldwin 



The Automated Testing Framewnrk 

Write a test, any test, and you'll have con¬ 
tributed to the overall stability and quality 
of the system. Even a very basic test can 
reveal problems. By Kristof Provost 




Diagnosing Excess LDAP Cennectiens 

Using DTrace The author, who man¬ 
ages a CS department's big data cluster, 
recently faced an interesting issue that he 
diagnosed using DTrace. This is the back- 
story. By Benedict Reuschling 




Foundation Letter Spring cleaning. By George Neville-Neil 



New Faces of FreeBSD in this installment, the spotlight is on Kai Knoblic, 
who received his ports bit in February. By Dru Lavigne. 

We Get Letters! Hello, illustrious Master Sysadmin. By Michael W Lucas .. 

Events Calendar By Dru Lavigne & Anne Dickison. 



SVn Update it has been a topic richly debated in the community for years, 
but Address Space Layout Randomization has landed in FreeBSD 13. 

By Steven Kreuzer. 



Conference Report FOSDEM 2019 Recap. FOSDEM is a must-attend 
conference! We look forward to continued Foundation presence and ongoing 
sponsorship of the FreeBSD Developer Summit. By Deb Goodkin. 
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Donate to the Foundation! 

You already know that FreeBSD is an internationally 
recognized leader in providing a high-performance, 
secure, and stable operating system. It's because of 
you. Your donations have a direct impact on the Project. 

Please consider making a gift to support FreeBSD for the 
coming year. It's only with your help that we can continue 
and increase our support to make FreeBSD the high- 
performance, secure, and reliable OS you know and love! 

Your investment will help: 

• Funding Projects to Advance FreeBSD 

• Increasing Our FreeBSD Advocacy and 
Marketing Efforts 

• Providing Additional Conference 
Resources and Travel Grants 

• Continued Development of the FreeBSD 
Journal 

• Protecting FreeBSD IP and Providing 
Legal Support to the Project 

• Purchasing Hardware to Build and 
Improve FreeBSD Project Infrastructure 
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In much of the Northern Hemisphere, 
spring has sprung, and in spring, a 
young developer's fancy turns to 
debugging and testing. If you doubt 
this, then you need look no further 
than the three excellent articles in 
this issue to change your mind. 

A s proof, we offer: "Debugging with GDB" by 
John Baldwin, "The Automated Testing 
Framework" by Kristof Provost, and "Diagnosing 
Excess LDAP Connections Using DTrace" by 
Benedict Reuschling. While it's true that many 
developers continue to debug with printfQ, you 
don't have to live that way, and John Baldwin gives 
us a tour of the latest version (8.3) of gdb, the GNU 
debugger, and shows how it can save hours of frus¬ 
tration in tracking down bugs in our code. 
Preventing bugs in our code is best done with good 
tests, and that's exactly what Kristof Provost 
explains to us in his article. Rounding out the three 
is Benedict Reuschling's personal-experience/back- 
story article on using DTrace to debug problems on 
a live system. 

We have a fresh crop of columns for this spring 
issue as well. Dru Lavigne profiles Kai Knoblic, a 
new ports committer; Michael W Lucas answers 
some letters; and Steven Kreuzer tells us all that's 
new in the source tree in svn Update. Deb Goodkin, 
Executive Director of the FreeBSD Foundation, gives 
a report on FOSDEM, the first big, open-source con¬ 
ference for 2019, which drew over 5,000 open- 
source project participants from all over Europe and 
the world for a weekend of meetings, presenta¬ 
tions, and meetups in Belgium. Speaking of confer¬ 
ences, if you're interested in attending a FreeBSD- 
related event, you'll find a list in Events Calendar, 
curated by Dru Lavigne and Anne Dickison. 

We look forward to seeing as many readers as 
possible at BSDCan in Ottawa, Ontario, in mid-May. 
Most of the Editorial Board members will attend, so 
if you see us there, please stop to say "hi" and let 
us know how we're doing with FreeBSD Journal. 

George Neville-Neil 
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PBhugging 

with BDB on FreeBSD 

by John Baldwin 


Debuggers are tools used to inspect the state of processes in a 
system. For running processes, debuggers can examine the values 
of variables as well as control process execution. For processes that 
have terminated abnormally due to a bug, debuggers can parse a 
process core dump generated by the kernel to inspect 
the state of the process at the time of the crash. 



ost debuggers support a base set of features such as examining 
the value of global and local variables, generating stack traces, 
interrupting process execution via breakpoints, and controlling 
process or thread execution via stepping. This article focuses on 
some of the other features supported by modern versions of the GNU 
debugger (gdb) on FreeBSD. Some of these features are only supported in 
the latest version of gdb (8.3 at the time of writing), while other features 
are available in older versions. 

To get started, install an up-to-date version of gdb. The simplest way is 
to install the pre-built package by running pkg install gdb. 
Alternatively, gdb can be built from source via the devel/gdb port 
( https://www.freshports.org/devel/gdb ). 


The info proc Command 

The info proc command can be used to examine process state beyond 
memory and threads. By default, info proc provides basic information 
such as the process ID and command line of a process. However, addition¬ 
al information is available via subcommands including the list of open file 
descriptors via info proc files and the list of active memory map¬ 
pings via info proc mappings. In the first two examples, the com¬ 
mand wc /usr/src/bin/ls/ls .c is being executed under the debug¬ 
ger. Execution is paused inside the cnt function with the target file open. 

The first example shows the information provided by the basic command. The 
second example shows the list of open files and includes the offset of the file 
descriptor for Is .c showing how much of the file the wc process has read. 
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While all the information available via info proc is also provided by other 
utilities such as ps ( 1 ) ( https://vvvvvv.freebsd.org/cgi/man.cgi?query=ps(1) ).and 
procstat( 1) ( https://www.freebsd.org/cgi/nnan.cgi?query=procstat(1) ), the info 
proc command permits a user to access them from within the debugger with¬ 
out having to open a separate window. These commands can also be used on 
a cross-debugger hosted on a non-FreeBSD OS while examining a core dump. 

More detailed information on the info proc command and its subcommands 
can be found in the Process Information section ( https://sourceware.org/ 
gdb/current/onlinedocs/gdb/Process-Information.html ) of the Debugging with 
GDB manual ( https://sourceware.org/gdb/current/onlinedocs/gdb/ ). 

Example 1: info proc 

(gdb) Info proc 

process 85146 

cmdline = '/usr/bin/wc /usr/src/bin/ls/ls.c' 

cwd = '/usr/home/john' 

exe = '/usr/bin/wc' 

Example 2: info proc files 

(gdb) info proc files 

process 85153 


Open 

files: 




FD 

Type 

Offset 

Flags 

Name 

text 

file 

- 

r- 

/usr/bin/wc 

ctty 

chr 

- 

rw- 

/dev/pts/20 

cwd 

dir 

- 

r- 

/usr/home/john 

root 

dir 

- 

r- 

/ 

0 

chr 

0xac82 

rw- 

/dev/pts/20 

1 

chr 

0xac82 

rw- 

/dev/pts/20 

2 

chr 

0xac82 

rw- 

/dev/pts/20 

3 

file 

0x63dd 

r- 

/usr/src/bin/ls/ls.c 


Intercepting System Calls 

GDB supports a special class of breakpoints called catchpoints. Catchpoints 
permit the user to pause execution when certain types of events occur dur¬ 
ing execution. One of the catchpoint types supported by GDB is a system 
call catchpoint. A system call catchpoint pauses execution on entry and exit 
from system calls. 

System call catchpoints are created using the catch syscall com¬ 
mand. If no arguments are specified, execution will pause on entry and exit 
from all system calls. More specific catchpoints can be defined by specifying 
a list of system calls as arguments to the command. System calls can be 
named either by name or number. For example, catch syscall write 
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sets a catchpoint that will pause execution on entry and exit from 
the write ( 2 ) ( https://www.freebsd.org/cgi/man.cgi?query=write(2) ) 
system call. 

Once a system call catchpoint has been created, it can be 
managed using other commands used with breakpoints. The 
info breakpoints command will list catchpoints along with other 
breakpoints. Catchpoints are removed via the delete command. 
Example 3 intercepts a write ( 2 ) ( https://wvvw.freebsd.org/cgi/ 
man.cgi?query=write(2) ) system call of a Is (1) 
( https://wvvw.freebsd.org/cgi/man.cgi?query=ls(1) ) process. 


Example 3: Catching a System Call 

% gdb -q --args /bin/ls -1 /bin/sh 

Reaciing symbols from /bin/ls... 

Reading symbols from /usr/lib/debug//bin/ls.debug... 

(gdb) catch syscall write 
Catchpoint 1 (syscall 'write' [4]) 

(gdb) info breakpoints 

Num Type Disp Enb Address What 

1 catchpoint keep y syscall "write" 

(gdb) run 

Starting program: /bin/ls -1 /bin/sh 

Catchpoint 1 (call to syscall write), _write () at _write.S:3 
3 PSEUDO(write) 

(gdb) c 
Continuing. 

-r-xr-xr-x 1 root wheel 168880 Nov 17 17:38 /bin/sh 

Catchpoint 1 (returned from syscall write), _write () at _write.S:3 
3 PSEUDO(write) 

(gdb) c 
Continuing. 

[Inferior 1 (process 24875) exited normally] 


For FreeBSD, GDB recognizes compatibility system calls. Catching a 
system call by name that has compatibility system calls for older ver¬ 
sions will catch all versions of that system call. For example, several sys¬ 
tem calls moved to new numbers in FreeBSD 12 due to changes in 
struct stat. The existing system calls continue to use the old layout 
of struct stat but were renamed to add a freebsdll_ prefix. 
When one of these system calls is caught by name, GDB catches both 
since applications might use either version. In Example 4, catching the 
f stat ( 2 ) ( https://wvvw.freebsd.org/cgi/man.cgi?query=fstat(2) ) system 
call registers catchpoints for both versions. 
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Example 4: Catch fstat(2) 

(gcib) catch syscall fstat 

Catchpoint 1 (syscalls 'freebscill_fstat' [189] 'fstat' [551]) 

(gcib) info breakpoints 

Num Type Disp Enb Aciciress What 

1 catchpoint keep y syscalls "freebsdll_fstat^ fstat" 


Debugging Forks 

Many programs create new processes via the fork ( 2 ) 
( https://wvvw.freebsd.org/cgi/man.cgi?query=fork(2) ) and vf ork (2 ) 
( https://www.freebsd.org/cgi/man.cgi?query=vfork(2) ) system calls. GDB 
provides several facilities for working with child processes created as a 
result of a fork. The following examples will demonstrate these facilities 
using the test program from Listing 1. 


Listing 1: forktest.c 

#include <sys/types.h> 

#include <sys/wait.h> 

#include <err.h> 

#include <stdio.h> 

#include <stdlib.h> 

#include <unistd.h> 

int 

main(void) 

{ 

pid_t pid, wpid; 

pid = fork(); 
if (pid == -1) 

err(1, "fork"); 
if (pid == 0) { 

printf("I'm in the childXn"); 
exit(1); 

} 

printf("I'm in the parentXn"); 
wpid = waitpid(pid, NULL, 0); 
if (wpid < 0) 

err(1, "waitpid"); 

return (0); 

} 


March/April 2019 


7 











Fork Follow Mode 

When a debugged process forks, GDB has to choose which process to 
continue debugging: the original (parent) process, or the new (child) 
process. By default, GDB follows the parent process letting the child 
process run freely after a fork as in Example 5. Note that the child process 
writes its output to the console and exits even while the parent process is 
paused in the debugger. 

Example 5: Following the Parent 

(gdb) start 

Temporary breakpoint 1 at 0x201354: file forktest.c, line 13. 

Starting program: /usr/home/john/work/johnsvn/test/forktest/forktest 

Temporary breakpoint 1 , main () at forktest.c:13 

13 pid = fork( ) ; 

(gdb) n 

[Detaching after fork from child process 25302] 

I'm in the child 

14 if (pid == -1) 

(gdb) p pid 

$1 = 25302 
(gdb) n 

20 printf("I'm in the parentXn"); 

(gdb) c 
Continuing. 

I'm in the parent 

[Inferior 1 (process 25297) exited normally] 


GDB uses the follow-fork-mode setting to determine which process 
to follow after a fork. To follow the child process instead of the parent, 
use the "child" setting. To restore the default behavior, use the "parent" 
setting. The setting is changed via the set follow-fork-mode com¬ 
mand. The show follow-fork-mode command displays the current 
setting. Example 6 executes the test program again but follows the child 
process instead of the parent process. 


Example 6: Following the Child 

(gdb) set follow-fork-mode child 
(gdb) start 

Temporary breakpoint 1 at 0x201354: file forktest.c, line 13. 
Starting program: /usr/home/john/work/johnsvn/test/forktest/forktest 

Temporary breakpoint 1 , main () at forktest.c:13 
13 pid = fork(); 
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Example 6 continued from page 8 


(gdb) n 

[Attaching after LWP 100857 of process 26342 fork to child LWP 101958 of process 26347] 
[New inferior 2 (process 26347)] 

[Detaching after fork from parent process 26342] 

[Inferior 1 (process 26342) detached] 

I'm in the parent 

[Switching to LWP 101958 of process 26347] 
main () at forktest.c:14 
14 if (pid == -1) 

(gdb) p pid 
$1 = 0 
(gdb) n 

17 printf("I'm in the childXn"); 

(gdb) c 
Continuing. 

I'm in the child 

[Inferior 2 (process 26347) exited with code 01] 


Detach on Fork 

In addition to deciding which process to follow after a fork, GDB provides 
a choice on how to treat the non-followed process. By default, GDB 
detaches from the non-followed process allowing it to run freely after the 
fork. The detach-on-fork setting can be changed to "no" to change 
this behavior. When it is set to "no", GDB stays attached to both process¬ 
es and leaves both processes paused after a fork. 

To manage the two processes, GDB's multiprocess support 
(https://sourceware.org/gdb/current/onlinedocs/gdb/lnferiors-and- 
Programs.html#lnferiors-and-Programs ) is used. In GDB terminology, each 
process is associated with an "inferior". The info inferiors com¬ 
mand is used to list the active inferiors. The inferior command is used 
to switch between inferiors. Threads from separate processes are also dis¬ 
played in the info threads command. Switching to a thread in differ¬ 
ent inferior is another way to switch inferiors. After a fork, the inferior of 
the followed process is set as the current inferior. 

Example 7 provides one more example of the test program. This time, 
detach-on-fork is disabled leaving both processes paused after the 
fork. The default fork follow mode is used, so GDB remains focused on 
the parent process after the fork. Note that the child process is stopped at 
the first instruction after the fork. 


Example 7: Staying Attached after Fork 

(gdb) set detach-on-fork off 
(gdb) start 

Temporary breakpoint 1 at 0x201354: file forktest.c, line 13. 
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Example 7 continued from paye 8 

Starting program: /usr/home/john/work/johnsvn/test/forktest/forktest 

Temporary breakpoint 1 , main () at forktest.c:13 

13 pid = fork(); 

(gdb) n 

[New inferior 2 (process 26828)] 

14 if (pid == -1) 

(gdb) p pid 

$1 = 26828 

(gdb) info inferiors 

Num Description Executable 

*1 process 26823 /usr/home/john/work/johnsvn/test/forktest/forktest 

2 process 26828 /usr/home/john/work/johnsvn/test/forktest/forktest 

(gdb) inferior 2 

[Switching to inferior 2 [process 26828] 

(/usr/home/john/work/johnsvn/test/forktest/forktest)] 

[Switching to thread 2.1 (LWP 101425 of process 26828)] 

Reading symbols from 

/usr/home/john/work/johnsvn/test/forktest/forktest.debug...done. 

Reading symbols from /usr/lib/debug/lib/libc.so.7.debug...done. 

Reading symbols from /usr/lib/debug/libexec/ld-elf.so.1.debug...done. 

#0 _fork () at _fork.S:3 
3 PSEUDO(fork) 

Warning: the current language does not match this frame. 

(gdb) n 

main () at forktest.c: 14 
14 if (pid == -1) 

(gdb) p pid 
$2 = 0 

(gdb) info threads 

Id Target Id Frame 

1.1 LWP 100970 of process 26823 main () at forktest.c:14 

*2.1 LWP 101425 of process 26828 main () at forktest.c:14 

(gdb) c 
Continuing. 

I'm in the child 

[Inferior 2 (process 26828) exited with code 01] 

(gdb) thread 1 . 1 

[Switching to thread 1.1 (LWP 100970 of process 26823)] 

#0 main () at forktest.c: 14 
14 if (pid == -1) 

(gdb) c 
Continuing. 

I'm in the parent 

[Inferior 1 (process 26823) exited normally] 
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Catching Forks 

GDB provides one final set of tools to aid with debugging forking 
processes: a set of catchpoints for events related to forks. The 
catch fork command installs a catchpoint for fork invocations that are 
not from vf ork ( 2 ) ( https://www.freebsd.org/cgi/man.cgi?query=vfork(2) ). 
The catchpoint triggers when the followed process returns from forking. 
The catch vf ork command installs a catchpoint for fork invocations 
from vfork(2). Finally, the catch exec command installs a catchpoint 
for returns from the exec family of system calls. Example 8 follows a shell 
process that forks a child process to execute a command. 


Example 8: Catching Fork and Exec 
% gdb -q /bin/sh 
(gdb) catch fork 
Catchpoint 1 (fork) 

(gdb) catch exec 
Catchpoint 2 (exec) 

(gdb) set follow-fork-mode child 
(gdb) run 

Starting program: /bin/sh 
$ Is -1 /dev/null; exit 

Catchpoint 1 (forked process 27644), _fork () at _fork.S:3 
3 PSEUDO(fork) 

(gdb) c 
Continuing. 

[Attaching after LWP 100469 of process 27639 fork to child LWP 101734 of 
process 27644] 

[New inferior 2 (process 27644)] 

[Detaching after fork from parent process 27639] 

[Inferior 1 (process 27639) detached] 

process 27644 is executing new program: /bin/ls 

Thread 2.1 hit Catchpoint 2 (exec'd /bin/ls), .rtld_start () 
at /usr/src/libexec/rtld-elf/amd64/rtld_start.S:33 
33 xorq%rbp,%rbp# Clear frame pointer for good form 

(gdb) c 
Continuing. 

crw-rw-rw- 1 root wheel Oxf Feb 2 18:00 /dev/null 
[Inferior 2 (process 27644) exited normally] 


For more information on using GDB to debug forks, see the Debugging 
Forks ( https://sourceware.org/gdb/current/onlinedocs/gdb/Forks.html ) chap¬ 
ter of the GDB manual. 
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Debugging C++ STL Classes 

For some data types, the raw layout of a data structure might not corre¬ 
spond to the use and representation of that structure in the source code. 
This can be especially true of C-n- Standard Template Library (STL) classes. 
To aid with inspecting these structures, GDB permits python scripts to pro¬ 
vide two types of helper classes: pretty printers (https://sourceware.org/ 
gdb/current/onlinedocs/gdb/Pretty-Printing-APLhtml#Pretty-Printing-API) and 
xmethods (https://sourceware.org/gdb/current/onlinedocs/gdb/Xmethods-ln- 
Python.html#Xmethods-ln-Python) . 

Pretty printers override the default display of objects by the print com¬ 
mand. Each pretty printer is associated with one or more C-n- classes. They 
can also be associated with templated classes. For example, a pretty printer 
for std:: vector can display the contents of the vector as an array. 

Xmethods permit python scripts to simulate the effects of inlined C-i-i- 
class methods. When evaluating expressions, GDB will call functions 
defined in the debugged program if needed to evaluate an expression. 

This includes calling C-n- operator overloading functions. However, if 
methods are inlined, as is common with templated classes, there is no dis¬ 
crete function symbol for GDB to call. As a result, attempting to use these 
functions or operators in an expression fails. Xmethods can be used to 
bridge this gap. For example, an xmethod can be used to provide 
operator [ ] for std;; vector objects permitting a user to directly 
index a vector with the same syntax used in the original C-i-i- source. 

The LLVM C-i-i- library used by FreeBSD does not include a set of python 
scripts providing pretty printers and xmethods for commonly-used C-n- STL 
classes. However, an initial set of scripts are available at 
( https://github.com/bsdjhb/libcxx-gdbpy ). At the time of writing, these 
scripts have limited support for std::string, std: ;unique_ptr, and 
std;;vector. Examples 9 and 10 compare examining a std::vector of 
integers without and with these scripts installed. These python scripts are 
included by the devel/gdb port by default in versions 8.2.1_1 and newer. 


Example 9: std: : vector Without Python Scripts 

(gdb) p vector 

$1 = {<std::_1::_vector_base<int, std::_1::allocator<int> » = 

{<std::_1::_vector_base_coiTimon<true» = {<No data fields>}, _^begin_ = 

0x800244000, 

_end_ = 0x80024400c, 

_end_cap_ = {<std::_1::_compressed_pair_elem<int*, 0, false» = { 

_^value_ = 0x800244010}, 

<std::_1::_compressed_pair_elem<std::_1: :allocator<int>, 1, true» = 

{std::_1: :allocator<int» = {<No data fields>}, <No data fields>}, <No data fields>}}, <No data fields>} 
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Example 9 continued frem paoe 12 

(gdb) p vector [1] 

Could not find operator[]. 

Example 10: std::vector With Python Scripts 
(gdb) p vector 

$1 = std::vector of length 3 = {4^ 5, 6} 

(gdb) p vector[1] 

$2 = 5 



Cross Debugging with a System Root 

The GDB package from ports is built by default as a cross-debugger. This 
means that it is able to examine binaries and core dumps from other 
architectures and other operating systems. For example, one can examine 
a process core dump from an embedded FreeBSD ARM system using a 
GDB process on a faster x86 host. Another use case is debugging the ker¬ 
nel of a remote machine over a serial connection. 

When cross-debugging a self-contained binary such as a static binary or 
monolithic kernel, GDB is able to find all of the information it needs from 
the binary. However, when debugging a binary that depends on other 
binaries such as shared libraries or kernel modules, GDB needs to be able 
to find these other binaries. Normally GDB looks for these binaries on the 
host where it is being run, which works fine when debugging a native 
process or core dump. However, when cross-debugging, GDB needs to be 
able to access these additional binaries. This can be solved by using a sys¬ 
tem root. 

A system root is a copy of the shared binaries from a system stored at 
an alternate directory. Often a system root may contain a full system 
installation image. If the system root contains headers and libraries used 
by compilers, a cross-compiler can use the system root to compile binaries 
for the alternate system. Both GCC and clang use the —sysroot flag 
to instruct the compiler to look for headers and libraries inside of a system 
root. In GDB, a system root is indicated by setting the sysroot variable 
to the path of the system root. GDB will then look for shared libraries and 
kernel modules under that system root rather than in the host's root file 
system. 

As an example, suppose a process run on a Raspberry Pi crashed gener¬ 
ating a core dump. One can take the SD card from the Raspberry Pi and 
insert it into an x86 machine. The SD card can then be mounted, and the 
core dump can be examined on the x86 machine. GDB just needs to be 
told to use the mount point of the SD card as the system root. Example 
11 uses this to look at an ARM core dump running on an x86 host. In this 
case, the SD card from the Raspberry Pi was mounted at /mnt. 
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Example 11: Examining an ARM Core Dump on an x86 Host 
> gdb -q sigframe 

Reading symbols from sigframe...Reading symbols from 
/mnt/home/john/work/johnsvn/test/sigframe/sigframe.debug...done. 
done. 


(gdb) set sysroot /mnt 
(gdb) core-file sigframe.core 
[New LWP 100086] 

Core was generated by "./sigframe'. 
Program terminated with signal SIGABRT, 
#0 thr_kill () at thr_kill.S:3 
3 RSYSCALL(thr_kill) 

(gdb) info sharedlibrary 
From To Syms Read 

0x20092000 0x201e516c Yes 

0x20016000 0x20030ef4 Yes 


Aborted. 


Shared Object Library 
/mnt/lib/libc.so.7 
/mnt/libexec/ldelf.so.1 


If you forget to set the sysroot variable before loading the core file, you 
can still set it after the core file is loaded. GDB will look for shared 
libraries under the new system root automatically after it is changed. 

The sysroot variable also works when debugging a remote FreeBSD ker¬ 
nel. GDB will look for kernel modules and their associated debug infor¬ 
mation under the system root. This can be useful even when the target 
machine is the same architecture as the host but is running an operating 
system or operating system version different from the host. ■ 



Jihn lallwii is a systems software developer. 
He has directly committed changes to the FreeBSD 
operating system for nineteen years across various 
parts of the kernel (including x86 platform support, 
SMP, various device drivers, and the virtual memory 
subsystem) and userspace programs. In addition to 
writing code, John has served on the FreeBSD core 
and release engineering teams. He has also con¬ 
tributed to the GDB debugger and LLVM. John lives 
in Concord, California with his wife, Kimberly, and 
three children: Janelle, Evan, and Bella. 
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There are several reasons to write tests, and if you only take 
away one idea from this article, it is that you should be writing 
tests because it's good for the soul. Okay, that might be a lie, 
but it's certainly good for the project. 


D espite its importance, testing can be boring and repetitive 
work. Most of us are attracted to development work because 
we like building things more than we like breaking them. 
Fortunately, we have machines that are good at doing boring, 
repetitive work without ever asking for a raise or time off. 

Automating tests has many advantages. It means repeatable 
tests. It means we know exactly what we are and are not testing, 
and when something goes wrong, we can usually easily reproduce 
the problem. Very few things make developers as happy as a bug 
report with a reproducible test case. 

It also means it is easier to run the tests regularly, which means 
we more quickly discover things have broken. Users, it turns out, 
don't like it when their software suddenly explodes. Instead, they 
want it to be boring and predictable. Developers get to keep the 
exciting surprises to themselves. 


How Do FreeBSD Tests Work? 

Like all great artists, the FreeBSD developers stole/borrowed code 
from elsewhere. In this case, the source of inspiration was NetBSD. 
ATF, or the Automated Testing Framework, started out as a Google 
Summer of Code project. It was initially imported into NetBSD in 
2007 and imported into FreeBSD in 2012. 

ATF is the framework used to write tests. Another NetBSD proj¬ 
ect, Kyua, runs the tests and gathers up results. When running 
tests, you'll interact mostly with Kyua. When writing tests, you'll 
usually interact with ATF. It's possible to write tests without ATF 
and still have Kyua execute them, but we won't discuss that here. 
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Why Would I Want to Write a Test? 

Good question, glad you asked! 

There are several reasons to write tests, and if you only take one idea 
away from this article, it should be that you should be writing tests 
because it's good for the soul. Okay, that might be a lie, but it's certainly 
good for the project. 

Tests ensure that functionality you rely on doesn't break. FreeBSD runs 
its test suite very frequently. The results of the test run for amd64 can be 
found on https://ci.freebsd.orq/job/FreeBSD-head-amd64-test/ . 

Writing a test case is also a great way to help a developer fix your pet 
bug. A short test case that demonstrates the problem is often the key to 
getting bugs fixed. It not only conclusively demonstrates that there is a 
problem, but it also clearly explains how to reproduce it, which makes 
fixing it much easier. 

When working on pf issues, I often find I spend much more time trying 
to understand the setup and trying to reproduce the issue than I do on 
the debugging and fixing. When bugs don't get fixed, it's often because 
they can't be reproduced. 

If you've got a pet patch you're trying to get committed, adding a test 
can also be helpful. It demonstrates the problem you're fixing or the new 
functionality you're adding. This serves both to convince people that this 
change is a good idea and as an aid in reviewing the patch. After all, a 
developer who commits a patch for you will have to fix any problems it 
might cause as well. Demonstrating that your code is tested will make it 
that much more likely that your patch will be included. 

How to Use Tests 

The tests, if installed, can be found in /usr/tests on your system. If 
you've also got a source tree, you can find the source code for the tests 
in /usr/src/tests, or in a subdirectory of the application, library or 
other they test. For example, tests for /bin/sh can be found in 
/usr/src/bin/sh/tests; pfctl tests can be found in 
/usr/src/sbin/pfctl/tests. Tests for pf itself can be found in 
/usr/src/tests/sys/netpfil/pf. 

To run the tests, you'll want to install Kyua ('sudo 
pkg install kyua')- You can then issue 'kyua test' in 
/usr/tests. Many of the tests will want to be run as root: 



/usr/tests % sudo kyua test 

sbin/growfs/legacy_test:main -> passed [4.710s] 
sbin/devd/client_test:seqpacket -> passed [0.015s] 
sbin/devd/client_test;stream -> passed [0.015s] 
sbin/dhclient/option-domain-search_test:main -> passed [0.005s] 
sbin/pfctl/macro;space -> passed [0.051s] 
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You can also run subsets of tests by running the command in the test 
subdirectory: 


usr/tests/bin/cat % kyua test 
cat_test:align -> passed [0.023s] 
cat_test:b_output -> passed [0.025s] 
cat_test:e_output -> passed [0.023s] 
cat_test:nonexistent -> passed [0.022s] 
cat_test:s_output -> passed [0.026s] 
cat_test:se_output -> passed [0.023s] 
cat_test:vt_output -> passed [0.029s] 



Alternatively, you can enumerate the list of tests using 'kyua list' 
and pick individual tests to run: 


/usr/tests % kyua test lib/csu/static/fini_test:dso_handle_test 
lib/csu/static/fini_test;dso_handle_test -> passed [0.002s] 


To get more information about tests, use 'kyua list -v' . The extra 
information can include things like required programs, required user, or a 
description. 

It goes without saying that this is something you'll only want to do on 
an otherwise unused system. By their very nature, tests risk exposing bugs, 
and some bugs are bad enough to bring the entire system down. 

Moreover, because the tests try to exercise all sorts of features of the sys¬ 
tem, it's quite possible they'll make configuration changes which can inter¬ 
fere with the intended use of the system. 

Don't run them without permission from the admin of the system. If that 
happens to be you, the requirement to submit a written application in trip¬ 
licate can probably be waived. 

So, when would you run those tests? Usually during development, but 
they can also serve as smoke test for a newly installed system. Passing the 
tests does not guarantee that the hardware works perfectly, but it does 
increase confidence. 

How to Write a Test 

When it comes to writing your own tests, there are really two different 
topics to discuss: the mechanics of how to write the test and then the 
judgement of what to test, and how to test it. 

We'll cover the mechanics first, briefly, as this is mostly documented in 
the atf (7) man page. Tests can be written as shell scripts or as C (or 
C-I-I-) code—which one is best depends on what is being tested. Usually 
the shell version ('man 3 atf-sh') is best suited to test entire applica¬ 
tions and the C/C-n-version ('man 3 atf-c, man 3 atf-c-i"i- ') is best 
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suited when testing libraries or (kernel) APIs. 

atf-sh tests are executed by atf-sh, so they need a 
#! /usr/bin/env atf-sh first line. This is done by the installation 
code though, so we don't need to worry about it here. They always 
contain a atf_init_test_cases function. This lists the test cases. 
Each test case will always have a head and body and may also have a 
cleanup function: 



atf_test_case tcl 
tcl_head() { 

... first test case's header ... 

} 

tcl_body() { 

... first test case's body ... 

} 

atf_test_case tc2 cleanup 
tc2_head() { 

... second test case's header ... 

} 

tc2_body() { 

... second test case's body ... 

} 

tc2_cleanup() { 

... second test case's cleanup ... 

} 

... additional test cases ... 

atf_init_test_cases() { 
atf_add_test_case tcl 
atf_add_test_case tc2 
... add additional test cases ... 

} 


Note how all test case functions have the same prefix. This prefix is 
passed to the test system in the atf_init_test_cases () function, 
allowing the framework to find them. The cleanup function can have 
any name, but usually naming it cleanup is a good idea. 

The head function is used to configure the test case. We can set the 
test description using 'atf_set "descr" "test description 
goes here" '. If this test can only be run as a specific user (i.e., root) 
or on specific architectures, we'd add 'atf_set require.user 
root' or 'atf_set require.arch amd64'. A full list of options 
that can be set here can be found in 'man 4 atf-test-case'. 

If you need a more specific test, this can always be implemented 
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using 'atf_skip'. For example, tests that require the VIMAGE kernel 
option to be set can do this: 



if 

fi 


"$(sysctl -i -n kern.features.vimage)" != 1 ]; then 
atf_skip "This test requires VIMAGE" 


One snag here is that constructs like that can't be used in the head. 
They'll need to go in the test body. Now that all of this setup and prepara¬ 
tion is out of the way, we can finally look at the test body. This is where 
we run the application or code we want to put through its paces. 

Most of the heavy lifting will be done by atf-check ( 1 ). It executes a 
command and can help us by checking that the exit status, standard out¬ 
put, and/or standard error match our expectations. For example, we might 
want to test that the false command really returns the expected exit 
status (-S exit; 1) and produces no output on either stdout (-o 
empty) or stderr (-s empty): 


atf-check -s exit:l -o empty -e empty /usr/bin/false 


We could also expect a signal from the test program with 
atf_expect_signal, or expect a timeout with atf_expect_time- 
out. 

Finally, if we've written a test for something we know to be broken, we 
can also add this information to the test. Using atf_expect_f ail we 
can indicate that we expect this test to fail. Traditionally we'd also include 
the bug (PR) number in this message. The test will still be executed, but 
the failure won't be counted as a failed test. 

C Tests 

Tests in C (or C-i-i-) have a similar structure, albeit with somewhat differ¬ 
ent syntax. 

Here's a short example: 


#include <atf-c.h> 

ATF_TC(tcl) ; 

ATF_TC_HEAD(tcl, tc) 

{ 

atf_tc_set_md_var(tc, "require.user", "root"); 

} 

ATF_TC_BODY(tcl, tc) 

{ 

ATF_CHECK_EQ(3, 2 + 1); 

} 
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ATF_TP_ADD_TCS(tp) 

{ 

ATF_TP_ADD_TC(tc s, tc1); 
return atf_no_error(); 

} 



The structure should be familiar after what we've covered already. 
We've got an entry point (atf_tp_add_tcs ()) that lists all of our 
test cases, and our test case has a head (atf_tc_head ()) that tells 
the test framework we need to be root to run this test. 

We may have lied about that last bit, because all our test body does 
is make sure that 2 + 1 equals 3. We'd generally expect that statement 
to be true, so this test should pass. 

Hooking It All Up 

Now that we know how to write tests, we still need to work out how 
we can actually run them. Fortunately, the makefiles do all of the hard 
work for us and we only have to write something like this: 


# $FreeBSD$ 

ATF_TESTS_SH += sh_example 
ATF_TESTS_C += c_example 

.include <bsd.test.mk> 


This would expect to find a 'sh_example.sh' file and 'c_example.c' 
file, build them, and install them as part of the 'buildworld' and 'install- 
world' make targets. The makefiles will also automatically create the 
required directory structure, and Kyuafile. 

How to Debug Your Tests 

A sad fact of life is that code is never perfect from the start (why else 
would we need tests?), and test code is no different. It's quite possible 
that your test won't do exactly what you expected it to do and the kyua 
output is typically not very helpful: 


# kyua test example 

example:example -> failed: atf-check failed; see the output 
of the test for details [0.017s] 


Fortunately, kyua keeps track of a lot more than just that, and if we 
ask nicely, we can get more information about the failed test, its envi¬ 
ronment, and the output it generated during the tests: 
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# kyua report —verbose example;example 
===> Execution context 

Current directory; /usr/tests/sys/netpfil/pf 
Environment variables; 

HOME=/root 

LANG=C 

LC_CTYPE=en_US.UTF-8 
LC_PAPER=en_GB.UTF-8 
LOGNAME=root 
MAIL=/var/mail/root 

PATH=/home/kp/bin;/usr/local/sbin;/sbin;/usr/local/bin;/ 
bin;/usr/bin;/usr/sbin;/home/kp/bin;/bin;/sbin;/usr/bin;/ 
usr/sbin;/usr/games;/usr/local/bin;/usr/local/sbin;/usr/ 
pkg/bin;/usr/pkg/sbin 
SHELL=/bin/csh 

SUDO_COMMAND=/usr/local/bin/kyua test example 

SUDO_GID=1001 

SUDO_UID=1001 

SUDO_USER=kp 

TERM=xterm-256color 

USER=root 

===> example;example 

Result; failed; atf-check failed; 

see the output of the test for details 
Start time; 2018-12-28T17;57;49.314738Z 
End time; 2018-12-28T17;57;49.332141Z 
Duration; 0.017s 

Metadata; 

allowed_architectures is empty 
allowed_platforms is empty 
description is empty 
has_cleanup = false 
is_exclusive = false 
required_configs is empty 
required_disk_space = 0 
required_files is empty 
required_memory = 0 
required_programs is empty 
required_user is empty 
timeout = 300 

Standard output; 

Executing command [ false ] 
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standard error: 

Fail: incorrect exit status: 1, expected: 0 
stdout: 

stderr: 



===> Failed tests 

example:example -> failed: atf-check failed; see the output of the test for details [0.017s] 
===> Suitimary 

Results read from /root/.kyua/store/results.usr_tests_sys_netpfil_j)f.20181228-175749-261396,db 

Test cases: 1 total, 0 skipped, 0 expected failures, 0 broken, 1 failed 

Start time: 2018-12-28T17:57:49.314738Z 

End time: 2018-12-28T17:57:49.332141Z 

Total time: 0.017s 


The most interesting things here are the standard output and stan¬ 
dard error logs. They tell us we were executing the command 
' false ', we expected error status 'O', but got ' 1' instead. In this 
case that's because our test is wrong. Naturally we'd expect "false" 
to exit with a status of 1 and not 0. 

For more complex atf-sh tests, it can be quite useful to include 
' set -X ' at the top of the test body. This will tell the shell to log 
everything it executes, giving a good overview of what commands were 
run and where the test stopped. 

What Is a Good Test? 

Knowing how to write the test is only half the battle. Knowing what to 
test is equally important. There are no hard rules about what makes for 
a good test, but there are a few tricks that can help to find good sce¬ 
narios to test. 

The easiest way is to start from a known bug. If you know some¬ 
thing's broken, there's value in writing a test for it, if only to ensure 
that this particular bug will never happen again. Quite often, bugs will 
cluster around specific features or particular sections of code, so varia¬ 
tions on that scenario may catch more bugs. 

Another valuable test case is to check validation code. A lot of code 
checks user input to ensure that it falls within expected ranges. Pick a 
bunch of values and ensure that they're rejected or accepted as appro¬ 
priate. Typically, very low or high values, as well as values just inside and 
just outside the valid range are good test points. 

For example, say a given configuration knob is supposed to accept 
values between 0 and 10 inclusive. A good test would be to ensure 
that -1000, -1, 11, and 1000 are rejected and that 0, 5 and 10 are 
accepted. 
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When thinking about tests, one naturally tends to think about 'bad' 
scenarios: bad input, bad configuration, bad everything. It's also useful to 
test a setup where everything works as expected. 

Finally, it's important to remember that having some tests is vastly better 
than having none at all. Don't let uncertainty about what to test stop you. 
Write a test, any test, and you'll have contributed to the overall stability 
and quality of the system. Even a very basic test can reveal problems. 

vnet 

Finally, a few words about my favorite tests: network stack and firewall 
tests. 

12.0 is the first release where vnet is enabled by default. I won't go into 
detail about vnet, that's a topic for a different article written by someone 
else. 

vnet virtualizes the network stack. If that gobbledygook doesn't mean 
anything to you, try this: vnet gives each jail its own network stack. 
Network interfaces can belong exclusively to that jail. It can set its own 
IP addresses without help from the host. It's even possible to use ipfw or 
pf from inside the jail. A jail starts to look a lot like a virtual machine that 
way. 

Long story short, it means that you can create network setups that'd 
usually require multiple machines all from the comfort of your laptop. In 
seconds. It means it's possible, and actually pretty easy, to write tests for 
the entire network stack. Speaking from my perfectly neutral and not-at- 
all-biased perspective as the pf maintained it means you should be writing 
tests for pf. 

Find examples in /usr/src/tests/sys/netpfil/pf. 

Other people have taken advantage of this example to write tests for 
IPSec, see /usr/src/tests/sys/netipsec. • 


is a freelance embedded software engineer specializing in network and 
video applications. He is a FreeBSD committer, maintainer of the pf firewall in FreeBSD, and a 
board member of the EuroBSDCon Foundation. Kristof has an unfortunate tendency to stumble 
into uClibc bugs and a burning hatred for FTP. Do not talk to him about IPv6 fragmentation. 
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[DIAGNOSING] 

EXCB88 LDAP 

CnniiBBtinns 



I manage our CS department's big data cluster and recently 
faced an interesting issue I was able to diagnose using 
□Trace. Here is the backstory. 



I had set up a couple of student-accessible FreeBSD machines 
for lab exercises. Students typically log into the cluster 
machines assigned to them using a combination of SSH and 
PAM, authenticating themselves against the department's 
LDAP server. The LDAP server provides all the information the stu¬ 
dents require, like checking whether a password is correct, what 
the home directory is, and what shell they're using. For certain sys¬ 
tems, students are not allowed access. These nodes have an 
AllowedUsers line in /etc/ssh/sshd_conf ig that lists the 
accounts that are permitted to log in. This setup has worked for a 
number of semesters without problems, and it is integrated into 
our installation scripts to let users log into the system from the 
moment it is installed. 

One day, in the middle of the semester, my office phone rang. A 
colleague from the IT department told me that they had received 
an unusually high number of requests coming from the big data 
cluster subnet hitting their two LDAP servers. He gave me a couple 
IP addresses of the worst offenders that had shown up in their 
logs. Some were creating multiple requests per second—a number 
that would typically be much less. Clearly, this was odd and should 
not have been happening at such a high rate. After hanging up, I 
concentrated my search on the machines he mentioned. Having a 
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short list of offending machines was already a good start, as it gave me 
the material I needed to focus my investigative efforts and also permit¬ 
ted a comparison with systems that exhibit normal behavior. 

First, I did a few basic checks: were there any processes running that 
should not be there, were there too many students working on the 
machine at the same time, thus creating network requests? Both 
answers turned out negative, as I was the only person logged into the 
system. Only the expected lab software was showing up in top( 1) 
and ps ( 1 ) outputs. Next, running netstat ( 1 ) did not show any 
unusual connections or open ports other than the ones that were sup¬ 
posed to be open on this host. The same was true for the other hosts 
in question, so I needed to dig deeper. 

I knew the problem somehow involved our LDAP servers and some 
kind of software running on the hosts connecting to them. But what 
kind of application was it (third-party or from the OS itself) and how 
could I get an overview of the number of connections to the LDAP 
servers for each application? Surely I could run tcpdump( 1 ) for a 
while and see traffic to the LDAP servers go by. It would take more 
work to get an overview of the initiating applications. I looked at the 
DTrace one-liners on the FreeBSD wiki [ https://wiki.freebsd.org/DTrace/ 
One-Liners ] and found one that would list the number of connections 
the local system was making to a remote IP address. 

sudo dtrace -n 'tcp:::send { @[args[2]->ip_daddr] = count(); 



When I ran it for a couple of seconds and then stopped the trace, I 
was presented with a two-column output. The first column showed 
the foreign IP address and the second listed the number of connections 
made during the trace time in ascending order. That was helpful, as I 
saw that the two IP addresses of our LDAP servers were showing up 
among them and there certainly had been a high number of connec¬ 
tions during the short while I ran the trace. Over time, these counts 
increased and the longer I let the DTrace one-liner run, the more probes 
would fire. Not every program was making a connection every second. 

Letting the script run for a couple of minutes provided a better 
overview of what was going on. The only piece of information missing 
was the actual application making the requests. I extended the script 
to also give me the executable name (called execname in DTrace lingo) 
like this: 

sudo dtrace -n 'tcp:::send { @[execname, args[2]->ip_daddr] = count(); }' 


The output now has three columns: the first has the name of the pro¬ 
gram or process initiating the connection (for example sshd or sudo). 
The second and third are the same as in the previous trace. Since I knew 
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the target IP address I was looking for (one of our two LDAP servers), I could 
also filter it more by using 

sudo dtrace -n 'tcp;;;send {@[execname, args[2]->ip_daddr == 

"IP.address.LDAP.server"] = count(); }' 


intr 

0 

5 

sshd 

0 

23 

intr 

0 

44 

postgres 

0 

63 

postgres 

0 

259 

sudo 

0 

301 

intr 

0 

380 

postgres 

0 

405 

intr 

0 

405 

intr 

0 

422 

sudo 

0 

441 

psql 

0 

475 

sshd 

0 

550 

sshd 

0 

560 

sshd 

0 

665 

postgres 

0 

1073 



Looking at the output, a new question arises: why is the ssh daemon 
sending so much TCP traffic to our LDAP servers? Of course, we're running 
this script from an SSH session, which means that a good amount of traffic 
could be generated by us running the script. I stopped the trace and dis¬ 
connected from the SSH session. After logging in from the server console, I 
confirmed using w( 1) that I was still the only user currently logged in. 

Then I re-ran the trace. To my surprise, sshd was still establishing a lot of 
connections, even though no one was using SSH at the moment. 

Compared to other processes on the system, sshd was responsible for the 
second-largest amount of connects, while the leading connection was to 
the local server IP address. This was fine and did not cause any problems, 
but why a constant barrage of requests to both LDAP server IPs when no 
one was actively using SSH sessions? A little further down the list, another 
thing that caught my eye was the sudo process. Of course, I was running 
my script using sudo, so I stopped the trace again and repeated it as the 
root user. As before, the number of sudo connections did not decrease. 

Both ssh and sudo were using the LDAP server—one for enabling cer¬ 
tain users to log into the system, while the other gave permissions to run 
certain local privileged actions. Remember, I had used this configuration for 
a number of years, but only recently had I received complaints from the IT 
department about these connections. And since this trace was showing just 
one machine, you can imagine the number of total requests the LDAP 
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servers were getting just from this subnet that contains at least 40 nodes with 
the same configuration (FreeBSD and Linux). Clearly, this needed to be 
addressed or we'd risk a permanent network ban for those machines, which 
would be bad news in the middle of the semester when students and 
researchers were using those machines for their lab work. 

Consulting with the IT department, they suggested that I activate 
nscd(8), the name service caching daemon. The idea was to use the 
caching service to provide results from the local cache rather than contacting 
the LDAP servers for each request directly. As the information returned from 
the LDAP servers would not change very often, a local cache should reduce 
the load on the servers. It would also increase the local request lookup per¬ 
formance. After reading the respective man pages for nscd, nscd.conf, 
and nsswitch.conf on FreeBSD, I activated nscd using 


# sysrc nscd enable=yes 


Before starting the nscd service, I added "cache" statements to 
/etc/nsswitch.conf to make it look like this: 


group; cache files Idap 
hosts: cache files dns 
networks; cache files 
passwd; cache files Idap 
passwd_compat; nis 
shells; files 
services; compat 
services_compat: nis 
protocols; files 
rpc: files 



The default /etc/nscd.conf has most if not all caches enabled, so there 
was no need to make any changes to this file. Then I started the nscd service 
using 


# service nscd start 


I activated this on the servers that were reported to be the most prominent 
abusers of the LDAP servers. Then, because other job-related things needed 
my time, I did not run a second trace to confirm that the requests were less¬ 
ening in number as a result of activating nscd. A week and a half went by 
without me thinking much about it. There is a lesson here: confirm that an 
issue has been solved before moving on to other things. Mentally switching 
back to a problem after a while can be difficult. 

In my case, the problem manifested again when I was not at work. On a 
Wednesday morning, our IT department shut off the network of the entire 
big data cluster because the number of requests to the LDAP server were so 
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severe that they were thought to be an internal Denial of Service attack. 
This caused one of the lab groups that was using the cluster nodes that 
morning to lose access. Only after the professor attending to that group, 
who was completely unaware of the network lockout, opened a ticket with 
the IT department asking for help, did they reestablish the network. The lab 
group was able to work again; however, the issue remained (and com¬ 
plaints were reaching my inbox, of course). At peak times, a single IP was 
responsible for 12.5% of total traffic to the LDAP servers. Not good, so a 
solution had to be found, and quickly. 

Apparently, nscd had not solved the problem as expected, so I had to 
get back to the analysis. When searching online for similar issues that other 
people might have encountered, I did not find anything directly related to 
my problem. However, I did find discussions about pam_ldap, which is the 
client that I was using on the cluster nodes to establish the connection to 
the LDAP servers. It turned out that the FreeBSD port had not received any 
updates since April 1, 2016, and according to some users, the software 
had been badly designed from the beginning. People recommended an 
alternative client, written and maintained by Arthur de Jong: 
nss-pam-ldapd [ https://arthurdejong.org/nss-pam-ldapd/ ]. 

Among other things, the port description (cat pkg-descr in 
/usr/ports/net/nss-pam-ldapd) listed "less connections to the 
LDAP server". This sounded exactly like what I was looking for, so I did a 
test install on one of the cluster nodes that had caused many of those 
requests. 

Not only was the nss-pam-ldapd software designed completely from 
scratch and integrated with a local caching service called nslcd, it was 
also well documented. Additionally, the software was maintained with the 
last update of the FreeBSD port in November 2018—the time of the writ¬ 
ing of this article. The changes could be implemented without a reboot of 
the system. A word of warning: errors in the files in /etc/pam.d or 
/etc/nsswitch.conf can lock you out of the system, even the root 
user. When that happens, you need to repair this in single-user mode or 
using a live CD. 

Once the changes had been put in place and the nslcd service had 
been started, I ran the DTrace script again. I found an entry for nslcd in 
the first column connecting to the LDAP server (I had disabled nscd earlier 
as it did not help). This was now the caching daemon making the calls, 
providing the results to the local services (sudo and ssh) asking for the 
information. The number of connections to the LDAP servers was greatly 
reduced in the DTrace output. On the server side, our IT department also 
confirmed that the traffic had gone down to normal levels again. Needless 
to say, I rolled out the nss-pam-ldapd service with nslcd on all the 
other machines. I also updated our install scripts to use nss-pam-ldapd 
instead of pam-ldap when a system is installed. Note there is also a ver¬ 
sion available with SASL support called nss-pam-ldapd-sasl. 
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The nice thing about this exercise was that I did not have to think 
much about writing the DTrace probes myself. I could simply access them 
from the library of handy DTrace one-liners on the FreeBSD wiki 
[ https://wiki.freebsd.org/DTrace/One-Liners ]. There are already a couple of 
good and ready-to-use one-liners there for various areas like storage, 
networking, and syscalls. With a bit of modification, it is fairly straight¬ 
forward to construct a probe that provides the information you are look¬ 
ing for to help diagnose the more difficult problems. • 


BeiiillGt Reuselillni joined the FreeBSD Project in 2009. After 
receiving his full documentation commit bit in 2010, he actively began 
mentoring other people to become FreeBSD committers. He joined the 
FreeBSD Foundation in 2015, where he is currently serving as vice pres¬ 
ident. Benedict has a Master of Science degree in Computer Science 
and is teaching a UNIX for software developers class at the University of Applied 
Sciences, Darmstadt, Germany. Together with Allan Jude, he is host of the weekly 
BSDNow.tv (http://BSDNow.tv) podcast. 
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BY DRU LAVIGNE 



of FreeBSD 


This column shines a spotlight on contributors who recently 
received a commit bit and introduces them to the Free¬ 
BSD community. In this installment, the spotlight is on 
Kai Knoblich, who received his ports bit in February. 



"the feeling 
of your first 
commit is 
priceless." 


Tell us a bit about yourself, your background, and your interests. 

• I was born in Hanover but have lived with my family in Nuremberg for 
a while now. I have been hearing impaired since birth, but thanks to 
modern technology, this can be dealt with. You can probably classify 
me as a cyborg since I have a Cl. 

When I was 10 years old, my Dad brought me an i286 that had 
been discarded by his company. That i286 was an Olivetti M290S as I 
recall, and it initiated my interest in computers. Over time, I realized 
that I also liked to code and learn new programming languages. I 
taught myself a lot during my free time after school. 

At work, I am a networkVsystem-engineer and am responsible for 
third-level matters. When I'm not in front of a computer/notebook, I 
like to read (nonfiction) books, climb, hike, and play retro video games 
from the 8-bit/16-bit era, including text/graphic adventures. 


How did you first learn about FreeBSD and what about FreeBSD 
interested you? 

• My first encounter with FreeBSD (which was release 5.3 at that time) 
was in 2005.1 already knew about FreeBSD but had not had an oppor¬ 
tunity to look at it up close. 

I had a project in which I set up a server that was used for develop¬ 
ment of web applications. The customer had already had positive expe¬ 
riences with FreeBSD and wanted to use FreeBSD on the new server. 

The challenge was that some components had to be specially config¬ 
ured. As a result, the Ports collection had to be used instead of the 
binary packages. I quickly realized how flexible working with the Ports 
collection was and above all, how well everything is documented. 

After the project was successfully completed, I got more involved 
with the universe of the BSDs. I had been a pure admin/user of 
FreeBSD for a while, which meant I switched between the releases by 
upgrading, was happy with the selection of ports, and in case of prob¬ 
lems, I found the answers either in the mailing lists or via search 
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engines. 

With more free time at the beginning of 2017, I got interested in the 
processes from the point of view of a developer. I asked myself questions 
like: Who does what at FreeBSD? What work is done to bring out new 
releases/ports? What are the next goals? And, can I get involved somehow? 
The last question particularly interested me, and I began to look more and 
more into the matter. 


How did you end up becoming a committer? 

• As I briefly mentioned earlier, I had been a passive user of open-source 
software over the years. I'd always thought that as soon as the time was 
right. I'd want to give back something to the community. 

At the beginning of 2018, I got a new notebook with freshly installed 
FreeBSD on it. I studied the Porter's Handbook and started initially to sub¬ 
mit patches for smaller ports. Not much later, I was already working on 
new ports and larger projects, such as net-mgmt/netbox, which depended 
on further packages. 

After NetBox became available in the Ports tree, I sent quite a lot of 
patches for many Django ports to cope with the aftermath of the Django 
1.8 to 1.11 transition. In July 2018, I was asked by Martin (miwi@) if I 
was interested in a commit bit. 

I decided to give it some thought, and a little later Jochen (joneurn®) 
and Tobias (tcberner®) also asked if a commit bit would be an option for 
me. I agreed in late autumn and at the beginning of February 2019 I had 
my commit bit. 


How has your experience been since joining the FreeBSD Project? 

Do you have any advice for readers who may also be interested in 
becoming a FreeBSD committer? 

• The welcome after I sent my introduction mail to the mailing list was 
great and the feeling of your first commit is priceless. At the moment, I 
can't give advice based on a lot of experience but I can offer some hints for 
getting started on the road to becoming a committer: 

• Pick an area (/doc, /ports, or /src) you enjoy. 

• Contribute clean and working patches via Bugzilla that do not need to 
be heavily revised by the committers. 

• Remember that the FreeBSD community is made up of volunteers and 
many have other priorities dealing with real life, family, and job. 

• And the most important thing: it should be fun. 


DRU LAVIGNE is a FreeBSD doc committer and the author of 
BSD Hacks and The Best of FreeBSD Basics. 
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Wo fJot letters 

by Michael W Lucas 



letters® 

freebsdjournal.c 


Hello, illustrious master sysadmin, 

Maybe something’s wrong with my servers, but I’m 
not sure. I get complaints, but when I look, everything’s 
fine. And the complaints are pretty weak—“it’s slow,” 

“it doesn’t work,” or “why do we even pay you people?” 
I don’t really know if there is anything wrong, but my gut 
says there might be. How do you track down such hazy 
problems? 

Thanks, 

Befuddled 


Dear Befuddled, 

The annoying thing about asymptomatic system failures is that 
they're asymptomatic—but no less real than the kind with noticeable 
symptoms. Some user makes a call, an actual voice call where they're 
spewing random words in some language from their food-hole and 
you're expected to parse that babble with your ears, when even 
Hollywood knows that sysadmins are artisanally optimized to receive 
information via their eyes and extrude alloyed sarcasm and results 
from their keyboard-callused fingertips. Any one of these users can at 
any time disrupt your meticulously assembled hallucination of what¬ 
ever problem you're working on and demand that you turn your 
three pounds of skull-pudding to the fact that their web browser jit¬ 
tered, actually jittered, when they played a cat video off the fileserver 
or they got a "File not found" error when they know durn well that 
they saved their proposal under that name just last night on their 
son's computer. 

The obvious solution—assigning every user who makes a voice call 
one of those nifty PDP-11 emulators as a desktop system until they 
learn enough about computing to be allowed near a machine with a 
monitor, like a Sinclair ZX-81—won't work. They'll only call more. 

But some errors are more mysterious. There's nothing you can 
point to. No spewing volcano of log messages, no cryptic PHP errors 
screeching about missing files or database timeouts, not even any 
ping failures. Something simply feels... wrong. 

You're descended from a long line of monkeys that survived the 
brutal savannah long enough to become parents before starving 
capybaras devoured them. (New archaeological evidence declares 
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that "torn apart by hyenas" was reserved for nobility.) Maybe your soft¬ 
ware understands shell script, but that aforementioned lump of skull¬ 
pudding hasn't been upgraded since the Paleolithic Era and doesn't have 
many tools to work with. 

The result is that when you arrive at your ergonomically hostile cubicle 
every morning, a part of your brain screams that you need to climb a 
tree right now. 

The only way to shut up that voice is data. 

Every computing organization has monitoring software, probably 
something like Icinga or whatever: boring, reliable, and consciously and 
deliberately limited. Whole meetings get wasted discussing what this 
monitoring should check, how often it should check, when it should 
alarm and when it should ignore detected problems. 

It's not that those problems don't occur. Maybe the monitoring checks 
the company's ERP system every minute. Every hour or so, it misses one 
check. Your organization decides to ignore those, because by the time 
some feeble human perceives the message and logs into the system the 
problem has evaporated. 

That intermittent alert happened. Maybe it only lasted five millisec¬ 
onds, but something failed. That lost check acts on your brain the way a 
rustling in the tall grass acted on your 200th-great-grandma. 

Something lurks out there. Your subconscious knows it. Is it a death by 
noble tigers? Or shall archaeologists digging up your remains declare 
"Wow. The marks on these bones resemble flamingo teeth" before scur¬ 
rying off to write a widely ridiculed thesis? 

You need to spend time on that intermittent alert. 

The world is full of monitoring tools. Each is limited in its own infuriat¬ 
ing way. Even your complex, all-inclusive, carefully tuned Icinga with all 
the trimmings and extra gravy has gaps, plus 
Sysadmin Rule #25 declares "All monitoring 
reduces to 'send an email to warn you email is 
broken.'" Look in those gaps. And this issue of 
the Journal has a whole bunch of information 
on monitoring and assessment tools. For any¬ 
thing involving the network, you should have 
netflow and SmokePing. If your network 
administrators don't have these tools. I've writ¬ 
ten a whole bunch of books about such topics and recommend you tre- 
buchet copies into the network department until they sense danger and 
install something useful. 

But which tools should you use? 

All of them. 

But not simultaneously. 

Every tool exposes something different. That's why we have so many. 
Explore DTrace—yes, I know, you're merely a puny sysadmin and you 


^^You don't have time 
to learn every new tool? 
Please. If my eyes rolled 
any harder, they'd fall 
out of my head and 
bounce off the table. 5 5 
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don't deal with system calls, but if your ancestors hadn't explored new 
things, you'd be capybara bait by the time you hit 20 so suck it up and 
dig in. 

You don't have time to learn every new tool? 

Please. If my eyes rolled any harder, they'd fall out of my head and 
bounce off the table. 

None of us have time. Look at me. I'm taking my time away from 
writing stuff that people will pay me for to answer your silly letter about 
the nebulous worries your tree-oriented subconscious is pushing 
upstairs, aren't I? The only reward I got for writing this column was the 
annual FreeBSD Journal board meeting held the night before BSDCan, 
and while previous years' programs included words like "opulent," 
"bacchanalia," and "sybaritic" the Journal's gone free and I suspect that 
descriptions of future meetings will feature the phrases "gruel," "alley," 
and "precautionary vaccination"—but you don't see me abusing my 
public platform in this very journal to gripe about these abusive 
changes, do you? No, this isn't griping. I'm better than that. This is 
merely a detailed example of what I'm not griping about. 

We never have time to learn new things. But learning new things is 
why we're in this nightmarish profession. The alternatives require wast¬ 
ing your dwindling supply of days deciphering the noises coming out of 
random people's food-holes. 

Pick tools that many people love. It's not that they're good tools; none 
of them are good, but these tools have a better pain/reward quotient 
than the rest. Play with them. Run each for a few hours, or a week. See 
what data each provides. Download some recommended DTrace scripts 
for your application software and see what it spends most of its time 
doing. Sure, some of what you learn will disturb anyone close enough 
to hear the screams with which you'll wrench yourself out of your mid¬ 
night nightmares, but you'll slowly assemble an awareness toolkit that 
lets you see the capybaras in the weeds. 

Develop your skills highly enough, and you too might rate death by 
hyena. 

PS: I commend the manners displayed in your salutation. Your parents 
raised you well. Not well enough to choose a better career, of course, 
but well. • 


Michael W Lucas ( https://mwl.io ) ’s newest book is FreeBSD Mastery: 
Jails, as well as 30-odd other titles like Absolute FreeBSD, PAM Mastery, 
and git commit murder. He’s shooting for death by tardigrade. Send 
your questions to letters@freebsdj ournal. com to be complimented in 
the most backhanded manner he can arrange. 
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BSD Events during May and June 2019 



BY DRU LAVIGNE & ANNE DICKISON 


FreeBSD Developers Summit • May 15 & 16 • Ottawa, Canada 

https://wiki.freebsd.org/DevSummit/201905 

BSDCan 2019 • May 15-18 • Ottawa, Canada 

https://www.bsdcan.org/2019/ BSDCan is a technical conference for people working 
on and with BSD operating systems and related projects. It is a developers conference with 
a strong focus on emerging technologies, research projects, and works in progress. It also features 
Userland infrastructure projects and invites contributions from both free software developers and com¬ 
mercial vendors. 






FreeBSD Hackathon and Bugbusting Session 
• May 20-22 • Kitchener-Waterloo, ON Canada 

https://hackmd.io/jrbtlbelQ1ahmqAjRHyUsQ# Join fellow FreeBSD develop¬ 
ers for a few days of collaborative software development or hardware hacking in 
Kitchener-Waterloo, Ontario, after BSDCan. 




FreeBSD Security Hackathon • June 8 & 9 • Vienna, Austria 

https://wiki.freebsd.org/Hackathon/201906 The first FreeBSD security-focused hackathon 
provides an opportunity to get together with FreeBSD hackers to discuss and work directly on fix¬ 
ing bugs, writing code, adding features, improving documentation, and working on ports, and 
much more. Besides hacking, there will be social events in the evenings. For the Monday Pentecost 
holiday there will be a sightseeing tour for interested attendees and their significant others. 


r; 


### 

rootconf 


Rootconf 2019 • June 21-22 • Bangalore, India 

https://rootconf.in/2019 / Rootconf is India's most well-known conference on 
DevOps and IT infrastructure. Rootconf attracts systems and operations engineers, 
and decision-makers in IT to share real world knowledge about building reliable 
systems. 
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It has been a topic richly debated in the community for years, 
but Address Space Layout Randomization has landed in FreeBSD 13. 
Depending on who you ask, this state-of-the-art mitigation technique 
will either keep your machine safer than encasing it in concrete or 
it’s a technology that has lived past its prime and only provides a false 
sense of security. As for my thoughts on the topic, as with politics, I 
don’t like to discuss security at the dinner table, but I will mention 
that I recently added WITH_PIE=yes into /etc/make.conf. 


A Implement Address Space Layout Randomization (ASLR)— 

https://svnweb.freebsd.orq/chanqeset/base/343964 

ith this change, randonnization can be enabled for all non-fixed map¬ 
pings. It means that the base address for the mapping is selected with 
a guaranteed amount of entropy (bits). If the mapping was requested to be 
superpage aligned, the randomization honors the superpage attributes. 

Although the value of ASLR is diminishing over time as exploit authors 
work out simple ASLR bypass techniques, it eliminates the trivial exploitation 
of certain vulnerabilities, at least in theory. This implementation is relatively 
small and happens at the correct architectural level. Also, it is not expected 
to introduce regressions in existing cases when turned off (default for now) 
or cause any significant maintenance burden. 

The randomization is done on a best-effort basis—that is, the allocator 
falls back to a first fit strategy if fragmentation prevents entropy injection. It 
is trivial to implement a strong mode where failure to guarantee the request¬ 
ed amount of entropy results in mapping request failure, but I do not consid¬ 
er that to be usable. 

Add WITH_PIE knob to build Position Independent 
Executables — https://svnweb.freebsd.orq/chanqeset/base/344179 
uilding binaries as PIE allows the executable itself to be loaded at a ran¬ 
dom address when ASLR is enabled (not just its shared libraries). With this 
change, PIE objects have a .pieo extension and INTERNALLIB libraries 
libXXX_pie.a. 

MK_PIE is disabled for some kerberosS tools. Clang, and Subversion, as 
they explicitly reference .a libraries in their Makefiles. These can be addressed 
on an individual basis later. MK_PIE is also disabled for rtid-elf because it is 
already position-independent using bespoke Makefile rules. Currently only 
dynamically linked binaries will be built as PIE. 

A Take steps towards multicore bhyve AMD support— 

https://svnweb.freebsd.orq/chanqeset/base/343075 

mm's CPUID emulation presented Intel topology information to the guest, 
but disabled AMD topology information and in some cases passed 
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through garbage. CPUID leaves 0x8000_001[de] were passed through to the 
guest, but guest CPUs can migrate between host threads, so the information pre¬ 
sented was not consistent. This could easily be observed with 'cpucontrol -i Oxfoo 
/dev/cpuctIO'. 

Slightly improve this situation by enabling the AMD topology feature flag and 
presenting at least the CPUID fields used by FreeBSD itself to probe topology on 
more modern AMD64 hardware (Family 15h-i-). Older stuff is probably less inter¬ 
esting. I have not been able to empirically confirm it is sufficient, but it should 
not regress anything either. 



Mask Spectre feature bits on AMD hosts — https://svnweb.freebsd.orq/ 
chanqeset/base/343166 

F or parity with Intel hosts that already mask out the CPUID feature bits that 
indicate the presence of the SPEC_CTRL MSR, do the same on AMD. 
Eventually we may want to have a better support story for guests, but for now, 
limit the damage of incorrectly indicating an MSR we do not yet support. 
Eventually, we may want a generic CPUID override system for administrators, or 
for minimum supported feature sets in heterogeneous environments with 
failover. That is a much larger scope effort than this bug fix. 



Add definitions for AMD Spectre/Meltdown CPUID information— 


htt 


3s://svnweb.freebsd.orq/chanqeset/base/343120 


N o functional change, aside from printing recognized bits in CPU identification. 

The bits are documented in 111006-B "Indirect Branch Control Extension"[1] 
and 124441 "Speculative Store Bypass Disable."[2] 



Notably missing — (left as future work): 

• Integration with hw.spec_store_bypass_disable and hw_ssb_active flag, which 
are currently Intel-specific 

• Integration with hw_ibrs_active global flag, which are currently Intel-specific 

• SSB_NO integration in hw_ssb_recalculate() 

• Bhyve integration (PR 235010) 



Implement per-CPU pmap activation tracking for RISC-V— 

https://svnweb.freebsd.orq/chanqeset/base/344108 

T his reduces the overhead of TLB invalidations by ensuring that we only inter¬ 
rupt CPUs that are using the given pmap. Tracking is performed in pmap_acti- 
vateO, which gets called during context switches: from cpu_throw(), if a thread is 
exiting or an AP is starting, or cpu_switch() for a regular context switch. For now, 
pmap_sync_icache() still must interrupt all CPUs. 



Implement transparent 2MB superpage promotion for RISC-V— 

https://svnweb.freebsd.orq/chanqeset/base/34410 6 

T he changes are largely modelled after amd64. arm64 has more stringent 
requirements around superpage creation to avoid the possibility of TLB conflict 
aborts, and these requirements do not apply to RISC-V, which like amd64, per¬ 
mits simultaneous caching of 4KB and 2MB translations for a given page. RISC- 
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Vs PTE format includes only two software bits, and as these are already con¬ 
sumed, we do not have an analogue for amd64's PG_PROMOTED. Instead, 
pmap_remove_l2() always invalidates the entire 2MB address range. 
pmap_ts_referenced() is modified to clear PTE_A, now that we support both 
hardware- and software-managed reference and dirty bits. Also fix 
pmap_fault_fixup() so that it does not set PTE_A or PTE_D on kernel mappings. 

A Provide userspace versions of do_cpuid() and cpuid_count() 
on i386 — https://svnweb.freebsd.orq/chanqeset/base/344118 

W hen generating PIC code, some older compilers cannot handle inline asm 
that clobbers %ebx (because %ebx is used as the GOT offset register). 
Userspace versions avoid clobbering %ebx by saving it to stack before execut¬ 
ing the CPUID instruction. 


STEVEN KREUZER is a FreeBSD Developer and Unix Systems 

Administrator with an interest in retro-computing and air¬ 
cooled Volkswagens. He lives in Queens, New York, with his 
wife, daughter, and dog. 


Thank you! 

The FreesBSD Foundation would like to 
acknowledge the following companies for 
their continued support of the Project. 
Because of generous donations such as 
these we are able to continue moving the 
Project forward. 

0 

FreeBSD 

FOUNDATION 

Are you a fan of FreeBSD? Help us give back to the 
Project and donate today! freebsdfoundation.org/donate/ 


Please check out the full list of generous community investors at 
freebsdfoundation.org/donors/ 



40 


FreeBSD Journal 














conference REP<9RT 

by Deb Goodkin 


FOSDEM 2019 recap 

Belgium is known for great beer, the 
best chocolate, and delicious waffles! 
It’s also the home of the largest open- 
source conference in Europe. FOSDEM 
2019 was held in Brussels, the capital 
of Belgium, February 2 and 3.1 was 
fortunate to attend again this year, 
along with the colocated FreeBSD 
Developer Summit held on February 1. 



T he Developer Sunnnnit had 20+ attendees. We held engaging dis¬ 
cussions on topics like bug busting, hackathons, Google Summer 
of Code project ideas, FreeBSD on Pinebooks, and more. These 
face-to-face opportunities with other contributors from the community 
are so valuable for the growth and longevity of the Project. People get 
inspired by others and go home energized to work on their own projects 
or join other projects they found interesting when discussed during the 
event or in the evenings over a beer. There was so much interest from 
more people wanting to attend this summit that next year we'll find a 
larger venue to allow more people to participate. 

FOSDEM started the next day at the Universite libre de Bruxelles. FOS¬ 
DEM is a free and noncommercial event organized by the community for 
the community. The goal is to provide free and open-source software 
developers and communities a place to meet. It's totally free! I believe 
there were over 8,000 attendees this year. It does make for very crowd¬ 
ed hallways. We were fortunate to get a stand again this year, and we 
were again located next to the lllumos project. Allan Jude tends to stand 
in between the two tables so he can talk ZFS to people who visit either 
project. As usual, we had a lot of FreeBSD volunteers helping to staff the 
table. These are people from all over Europe who are passionate about 
FreeBSD. 

There was a BSD Developer Room (devroom) that allowed up to 100 
people interested in the BSDs to meet and listen to 13 talks on the vari¬ 
ous BSD projects. I kicked off the devroom talks with my "25 Years of 
FreeBSD" presentation. You can find the recording here: 
https://fosclem.org/2019/schedule/event/25_years_of_freebsd/. 

I spent the rest of the day talking to people who stopped by our table. 
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learned about more companies using FreeBSD, and talked to 
other open-source community members. I was thrilled by the 
number of people who stopped by our table. Though we 
quickly ran out of our fabulous luggage identifiers and tape 
measures, we ended up giving away 1,000 stickers, "selling" 
40 t-shirts, and handing out lots of Getting Started with 
FreeBSD fliers. I witnessed many of the volunteers having 
engaging, in-depth technical discussions on specific parts of 
the kernel and features. 

I was very happy to see Shteryana Shopova there and get¬ 
ting back into FreeBSD. She ran EuroBSDCon 2014 in 
Bulgaria and worked on various parts of FreeBSD, focusing on network¬ 
ing. 

I was also able to attend Jonathan Looney's presentation called 
"Netflix and FreeBSD—Using Open Source to Deliver Streaming Video" 
in the main track. This was a powerful testimonial to the benefits of 
using FreeBSD. The room was packed and people came up to Jonathan 
afterward to ask questions. You can see his talk here: 
https://fosdem.org/2019/schedule/event/netflix_freebsd/. 

The second day was just as crazy and crowded though there were 
moments when you could actually walk through the hallways without 
getting elbowed. 

Besides connecting with more commercial users and getting people to 
play around with FreeBSD on their desktops, some of us within the 
FreeBSD community talked about having more FreeBSD tracks and 
workshops at more conferences in the next 12 months. This included a 
discussion on having a FreeBSD track at l inux.conf.au , running an Intro 
to FreeBSD workshop at All Things Open and 
womENcourage, just to name a few. 

In conclusion, FOSDEM is a must-attend conference for 
promoting FreeBSD. There were a few FreeBSD presentations 
in the main track, and we will encourage more people to 
submit talks in the future to grow its visibility. We look for¬ 
ward to having a continued Foundation presence there so 
companies and users can talk to us about what they are 
doing and what we can do to help, as well as continuing to 
sponsor the FreeBSD Developer Summit, and of course pro¬ 
vide all the cool swag. • 


DEB GOODKIN is the Executive Director of the FreeBSD Foundation. She’s thrilled 
to be in her 14th year at the Foundation and is proud of her hardworking and dedi¬ 
cated team. She spent over 20 years in the data storage industry in engineering 
development, technical sales, and technical marketing. When not working, you’ll 
find her on her road, gravel, or mountain bikes, training for a running race, hiking 
with her dogs, or playing with FreeBSD on her various systems. 
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